Large-Scale Collective Entity Matching

The difficult problem of Entity Matching (EM), i.e., determining whether different mentions of an entity refer to the same realworld object, has recently attracted a lot of attention from the Machine Learning (ML) research community. State-of-the-art solutions for EM are based on recent advances in ML, such as firstorder probabilistic models (e.g., Markov and Bayesian networks) and advanced probabilistic inference techniques. A key benefit of these ML tools comes from their purely collective nature: match evidence for related entities can be collectively reinforced into high-probability EM decisions. In addition, they provide a principled framework for imposing a probability distribution over possible EM results which could be useful in many settings (e.g., for user feedback). While such state-of-the-art ML approaches to EM have been shown to be very accurate in practice, they also typically require complex inference over very large model graphs; thus, their scalability to real-life datasets has remained a big challenge. Toward this end, in this work, we propose a principled framework to scale general, collective EM operators. Our framework is generic: it uses a black-box abstraction to incorporate any entity matcher. The main idea is to approximate the run of the entity matcher on the entire data set by: (1) running multiple instances of the matcher on several small subsets of the entities, and (2) Message-Passing, i.e., passing a judiciously-built message-set across the instances to control the interaction between different runs of the matcher. While the notion of communicating “blocks” of EM is not entirely new, our work is the first to carry out a complete, formal analysis of the above framework, and show that for a broad class of “well-behaved” entity matchers, the approach is provably sound. We also propose novel message-passing schemes for probabilistic EM tools that, as our results demonstrate, significantly improve EM recall without compromising soundness. Finally, we present experimental results demonstrating the effectiveness of our approach and its ability to scale to large real-life datasets.